Singular Value Decomposition and High-Dimensional Data

A data set with n measurements on p variables can be represented by an n × p data matrix X. In highdimensional settings where p is large, it is often desirable to work with a low-rank approximation to the data matrix. The most prevalent low-rank approximation is the singular value decomposition (SVD). Given X, an n × p data matrix, the SVD factorizes X as X = UDV ′, where U ∈ Rn×n and V ∈ Rp×p are orthogonal matrices and D ∈ Rn×p is zero except on its diagonal with diagonal entries in decreasing order. The best rank K approximation to X, X̂K , in both the Frobenius and operator norms is given by the first K right singular vectors and singular values of the SVD: X̂K = PK k=1 dkukv ′ k. The SVD of X is also closely related to the eigendecomposition of X ′X. Specifically, if UDV ′ is an SVD of X, then V (D′D)V ′ is an eigendecomposition of X ′X. Thus, the eigenvalues of X ′X are the squares of the singular values of X, and the eigenvectors of X ′X are the right singular vectors of X. To fully understand the implications of using the SVD in data-processing applications and classical multivariate analysis techniques such as principal components analysis (PCA), one must consider the behavior of the SVD when the elements of X are random.